Python

Packaging

Jan. 2023

Content

Python package: Concepts/Terminology
Developing a package
Build the package for its distribution
Publish the package on PyPi
Install your package from PyPi
Productionize your package/packaging

Concepts

# 1. CONCEPTS

Python script?
Python module?
Python package?
Python (standard) library?

Concepts: script & module

# 1. CONCEPTS

# in mod_add.py

def add(a, b):
  return a+b

# in my_script.py

def add(a, b):
  return a+b

result = add(30,12)
print(result)

Script

meant to be run
statements outside functions or class

Module

meant to be imported into scripts or other modules
defines functions, classes, ... for use in scripts

Concepts: module called in a script

# 1. CONCEPTS

# in mod_add.py

def add(a, b):
  return a+b

# in my_script.py

from mod_add import add


result = add(30,12)
print(result)

Script

meant to be run
statements outside functions or class

Module

meant to be imported into scripts or other modules
defines functions, classes, ... for use in scripts

Concepts: {script, module} all-in-one

# 1. CONCEPTS

Script

meant to be run
statements outside functions or class

Module

meant to be imported into scripts or other modules
defines functions, classes, ... for use in scripts

# in (exec.) mod_add.py

def add(a, b):
  return a+b

if __name__ == "__main__":
  result = add(30,12)
  print(result)

# in my_script.py

from mod_add import add


result = add(30,12)
print(result)

Concepts: package

# 1. CONCEPTS

a collection of related modules and/or sub-packages
can be an individual module, but usually a collection of them

Concepts: package (example)

# 1. CONCEPTS

codetiming package (source: Real Python)

Concepts: libraries

# 1. CONCEPTS

Collection of modules & packages

Python standard library
Third-party libraries

What are Python Libraries (source)

Concepts: Python standard library

# 1. CONCEPTS

a collection of modules & packages
come bundled with every installation of python
examples: os, sys, math, dateutil, email, json, ...

What are Python Libraries (source)

Concepts: third-party libraries

# 1. CONCEPTS

requests
Django
Flask
SQLAlchemy
numpy
scipy
pandas
pytest
pylint
flake8
psycopg2
...

Example 3rd-party libraries: Jupyter

# 1. CONCEPTS

Packaging Flow

# 2. PACKAGING FLOW

The Python package cycle (source: Python Packages)

Packaging Flow

# 2. PACKAGING FLOW

Develop & Configure a Python Package

- Create your Source Tree

- Package configuration file

Build & Distribute a Python Package

- Build Artifacts (source distribution, built distribution)

- Upload to the package distribution service (PyPi, Artifactory, ...)

- Binaries & Dependencies

Download & Install

History, Tips & Challenges

# 2. PACKAGING FLOW

Packaging Demystified (source: PyCon2021 Bernát)

Developing a package

# 3. DEMO PACKAGE

The Python package cycle (source: Python Packages)

Calculating \( \pi \)

\pi = 4 (1-\frac{1}{3}+\frac{1}{5}-\frac{1}{7}+\cdots)

The Gregory-Leibniz Series
(proof: @stanford.edu)

cf. Packaging Demystified (source: PyCon2021 Bernát)

# 3. DEMO PACKAGE

Developing a package

Create your Source Tree (folders, subfolders, files) with Poetry

➜  poetry new approx_pi
Created package approx_pi in approx_pi

➜  tree approx_pi/
approx_pi
├── README.md
├── approx_pi
│   └── __init__.py
├── pyproject.toml
└── tests
    └── __init__.py

2 directories, 4 files

# 3. DEMO PACKAGE

Developing a package

# pi_approx.py
from math import pi

def approximate_pi(iteration_count: int) -> float:
  sign, result = 1, 0.0
  for at in range(iteration_count):
    result += sign / (2 * at + 1)
    sign *= -1
  return result * 4

if __name__ == "__main__":
  approx_1, approx_2 = approximate_pi(300), approximate_pi(301)
  print(f"approx 300 {approx_1} with diff {approx_1 - pi}")
  print(f"approx 301 {approx_2} with diff {approx_2 - pi}")

Add functionalities to your main module

pi_approx.py

# 3. DEMO PACKAGE

Developing a package

➜ # Activating the virtual environment 
➜  poetry shell
Spawning shell within /home/patricme/.cache/pypoetry/virtualenvs/approx-pi-BE-Xuvxi-py3.9
➜  . /home/patricme/.cache/pypoetry/virtualenvs/approx-pi-BE-Xuvxi-py3.9/bin/activate
(approx-pi-py3.9)

➜ # Installing dependencies
➜  poetry install
Creating virtualenv approx-pi-BE-Xuvxi-py3.9 in /home/patricme/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Create a local environment & Install dependencies (Poetry)

# 3. DEMO PACKAGE

Developing a package

# Running a python script
(approx-pi-py3.9) ➜ python3 approx_pi/pi_approx.py
approx 300 3.1382593295155914 with diff -0.0033333240742017267
approx 301 3.1449149035588526 with diff 0.0033222499690594987

# Running a python module
(approx-pi-py3.9) ➜ python3 -m approx_pi.pi_approx
approx 300 3.1382593295155914 with diff -0.0033333240742017267
approx 301 3.1449149035588526 with diff 0.0033222499690594987

# 3. DEMO PACKAGE

Developing a package

Running your Python package

Adding a dependency from PyPi

(approx-pi-py3.9) ➜  poetry add cowsay
Using version ^5.0 for cowsay
Updating dependencies
Resolving dependencies... (9.2s)
Writing lock file
Package operations: 1 install, 0 updates, 0 removals
  • Installing cowsay (5.0)

# pi_approx.py
...

if __name__ == "__main__":
    approx_1, approx_2 = approximate_pi(300), approximate_pi(301)
    print(f"approx 300 {approx_1} with diff {approx_1 - pi}")
    print(f"approx 301 {approx_2} with diff {approx_2 - pi}")
    cowsay.cow(f"The difference between \nthe approximations is \n{'%.2E' % Decimal(approx_1-approx_2)}")

# 3. DEMO PACKAGE

# pyproject.toml
[tool.poetry]
name = "approx-pi"
version = "0.1.0"
description = ""
authors = ["Patrick Merlot <patrick.merlot@gmail.com>"]
readme = "README.md"
packages = [{include = "approx_pi"}]

[tool.poetry.dependencies]
python = "^3.9"
cowsay = "^5.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

# 3. DEMO PACKAGE

Developing a package

Updated config file after adding the new dependency (cowsay)

(approx-pi-py3.9) ➜ python3 -m approx_pi.pi_approx
approx 300 3.1382593295155914 with diff -0.0033333240742017267
approx 301 3.1449149035588526 with diff 0.0033222499690594987
  ______________________
 /                      \
| The difference between |
| the approximations is  |
| -6.66E-03              |
 \                      /
  ======================
                      \
                       \
                         ^__^
                         (oo)\_______
                         (__)\       )\/\
                             ||----w |
                             ||     ||

# 3. DEMO PACKAGE

Running your Python package

Developing a package

Build a distribution package

# 4. BUILD DISTRIBTUION PACKAGE

The Python package cycle (source: Python Packages)

Build the package's artifacts

(approx-pi-py3.9) ➜ poetry build
Building approx-pi (0.1.0)
  - Building sdist
  - Built approx_pi-0.1.0.tar.gz
  - Building wheel
  - Built approx_pi-0.1.0-py3-none-any.whl

the source distribution (sdist) & the built distribution (wheels)

# 4. BUILD DISTRIBTUION PACKAGE

Publishing your Python Package to PyPI

# 5. PUBLISH DISTRIBTUION PACKAGE

The Python package cycle (source: Python Packages)

Publishing your Python Package to PyPI

PyPI (pypi.org) is the default publishing target for Poetry.

In order to publish a package to PyPI, you will need to create an account. Go to the official registration page in your web browser.

# 5. PUBLISH DISTRIBTUION PACKAGE

Publishing your Python Package to PyPI

Token authentication is the recommended way to use your PyPI account in the command line.

You can use a single, automatically generated token instead of a username and password.

Enabling Token Authentication for PyPI.

# 5. PUBLISH DISTRIBTUION PACKAGE

Publishing your Python Package to PyPI

Add your API token to Poetry with this command

➜ poetry config pypi-token.pypi <your-api-token>
Using a plaintext file to store credentials

Publishing your Python Package to PyPI

(approx-pi-py3.9) ➜  poetry publish

Publishing approx-pi (0.1.0) to PyPI
 - Uploading approx_pi-0.1.0-py3-none-any.whl 100%
 - Uploading approx_pi-0.1.0.tar.gz 100%

# 5. PUBLISH DISTRIBTUION PACKAGE

Publishing your Python Package to PyPI

Check your published package in your PyPI projects, in the browser.

# 5. PUBLISH DISTRIBTUION PACKAGE

Your package is available for everyone to use via:
Pip, Poetry, Conda, Pipenv, ...

Installing a Python Package from PyPI

# 6. INSTALLING A PACKAGE

The Python package cycle (source: Python Packages)

Adding a dependency from PyPi

➜  poetry add cowsay
Using version ^5.0 for cowsay
Updating dependencies
Resolving dependencies... (9.2s)
Writing lock file
Package operations: 1 install, 0 updates, 0 removals
  • Installing cowsay (5.0)

➜  poetry add <any-python-package-available-on-pypi>

# 6. INSTALLING A PACKAGE

➜  poetry add approx-pi

Version control your code on (public) repo.

# 7. VERSION CONTROL

The Python package cycle (source: Python Packages)

Collaborate via Git repository providers

# 7. VERSION CONTROL

Version Control with Git

# Initialize your project (ONCE!)
git init

# Tell git where is the main repository (Github? Bitbucket? GitLab? ...)
# (ONCE!)
git remote add github https://github.com/Patechoc/approx_pi.git

# Specify the files that you want to add to the next "push" to the main repository
git add pyproject.toml approx_pi.py/ README.md

# Give a description to the changes you want to push
git commit -a -m "Minimum file structure for our python package"

# Push the changes that are committed to the main repository
git push

Tracking code changes (backup, undo, feature dev.)
Coding collaboration (simplified merge, branching)
Get started: www.w3schools.com/git/

# 7. VERSION CONTROL

Version Control with Git

Create useful .gitignore files for your project (source: Toptal)

# Created by https://www.toptal.com/developers/gitignore/api/python
# Edit at https://www.toptal.com/developers/gitignore?templates=python

### Python ###
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
#  Usually these files are written by a python script from a template
#  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
#   For a library or package, you might want to ignore these files since the code is
#   intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
#   However, in case of collaboration, if having platform-specific dependencies or dependencies
#   having no cross-platform support, pipenv may install dependencies that don't work, or not
#   install all needed dependencies.
#Pipfile.lock

# poetry
#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
#   This is especially recommended for binary packages to ensure reproducibility, and is more
#   commonly ignored for libraries.
#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
#   pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
#   in version control.
#   https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
#  and can be added to the global gitignore or merged into this file.  For a more nuclear
#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/

### Python Patch ###
# Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration
poetry.toml

# ruff
.ruff_cache/

# End of https://www.toptal.com/developers/gitignore/api/python
n

# 7. VERSION CONTROL

Don't save everything!

Testing

# 8. TESTING

Getting started with testing in Python (source: Real Python)

Testing

# 8. TESTING

https://tox.wiki

Code Security Check

# 9. SECURITY CHECK

Bandit is a tool designed to find common security issues in Python code.

To do this, Bandit processes each file, builds an AST from it, and runs appropriate plugins against the AST nodes. Once Bandit has finished scanning all the files, it generates a report.

Coding standards

# 10. CODING STANDARDS

Use isort for sorting the library imports (yes, imports have a suggested order).
Check the existence of undesired artifacts using Flake8 and Pylint.
Keep the code within the same style using Black.
Those tools can be configured to be PEP8 compliant. PEP8 — Python Enhancement Proposal, is a style guide that provides guidelines and best practices suggestions on how to write Python code.